Efficient Offline Policy Optimization with a Learned Model

MuZero Unplugged presents a promising approach for offline policy learning from logged data. It conducts Monte-Carlo Tree Search (MCTS) with a learned model and leverages Reanalyze algorithm to learn purely from offline data. For good performance, MCTS requires accurate learned models and a large number of simulations, thus costing huge computing time. This paper investigates a few hypotheses where MuZero Unplugged may not work well under the offline RL settings, including 1) learning with limited data coverage; 2) learning from offline data of stochastic environments; 3) improperly parameterized models given the offline data; 4) with a low compute budget. We propose to use a regularized one-step look-ahead approach to tackle the above issues. Instead of planning with the expensive MCTS, we use the learned model to construct an advantage estimation based on a one-step rollout. Policy improvements are towards the direction that maximizes the estimated advantage with regularization of the dataset. We conduct extensive empirical studies with BSuite environments to verify the hypotheses and then run our algorithm on the RL Unplugged Atari benchmark. Experimental results show that our proposed approach achieves stable performance even with an inaccurate learned model. On the large-scale Atari benchmark, the proposed method outperforms MuZero Unplugged by 43%. Most significantly, it uses only 5.6% wall-clock time (i.e., 1 hour) compared to MuZero Unplugged (i.e., 17.8 hours) to achieve a 150% IQM normalized score with the same hardware and software stacks. Our implementation is open-sourced at https://github.com/sail-sg/rosmo.Comment: ICLR202

Disturbance observer-based neural network control of cooperative multiple manipulators with input saturation

In this paper, the complex problems of internal forces and position control are studied simultaneously and a disturbance observer-based radial basis function neural network (RBFNN) control scheme is proposed to: 1) estimate the unknown parameters accurately; 2) approximate the disturbance experienced by the system due to input saturation; and 3) simultaneously improve the robustness of the system. More specifically, the proposed scheme utilizes disturbance observers, neural network (NN) collaborative control with an adaptive law, and full state feedback. Utilizing Lyapunov stability principles, it is shown that semiglobally uniformly bounded stability is guaranteed for all controlled signals of the closed-loop system. The effectiveness of the proposed controller as predicted by the theoretical analysis is verified by comparative experimental studies

Exploring the Potential of Integrated Optical Sensing and Communication (IOSAC) Systems with Si Waveguides for Future Networks

Advanced silicon photonic technologies enable integrated optical sensing and communication (IOSAC) in real time for the emerging application requirements of simultaneous sensing and communication for next-generation networks. Here, we propose and demonstrate the IOSAC system on the silicon nitride (SiN) photonics platform. The IOSAC devices based on microring resonators are capable of monitoring the variation of analytes, transmitting the information to the terminal along with the modulated optical signal in real-time, and replacing bulk optics in high-precision and high-speed applications. By directly integrating SiN ring resonators with optical communication networks, simultaneous sensing and optical communication are demonstrated by an optical signal transmission experimental system using especially filtering amplified spontaneous emission spectra. The refractive index (RI) sensing ring with a sensitivity of 172 nm/RIU, a figure of merit (FOM) of 1220, and a detection limit (DL) of 8.2*10-6 RIU is demonstrated. Simultaneously, the 1.25 Gbps optical on-off-keying (OOK) signal is transmitted at the concentration of different NaCl solutions, which indicates the bit-error-ratio (BER) decreases with the increase in concentration. The novel IOSAC technology shows the potential to realize high-performance simultaneous biosensing and communication in real time and further accelerate the development of IoT and 6G networks.Comment: 11pages, 5 figutre

Histone H3.3 and its proteolytically processed form drive a cellular senescence programme.

The process of cellular senescence generates a repressive chromatin environment, however, the role of histone variants and histone proteolytic cleavage in senescence remains unclear. Here, using models of oncogene-induced and replicative senescence, we report novel histone H3 tail cleavage events mediated by the protease Cathepsin L. We find that cleaved forms of H3 are nucleosomal and the histone variant H3.3 is the preferred cleaved form of H3. Ectopic expression of H3.3 and its cleavage product (H3.3cs1), which lacks the first 21 amino acids of the H3 tail, is sufficient to induce senescence. Further, H3.3cs1 chromatin incorporation is mediated by the HUCA histone chaperone complex. Genome-wide transcriptional profiling revealed that H3.3cs1 facilitates transcriptional silencing of cell cycle regulators including RB/E2F target genes, likely via the permanent removal of H3K4me3. Collectively, our study identifies histone H3.3 and its proteolytically processed forms as key regulators of cellular senescence.This is the author's accepted manuscript. The final version is available from NPG at http://www.nature.com/ncomms/2014/141114/ncomms6210/full/ncomms6210.html

Prenatal Progestin Exposure Is Associated With Autism Spectrum Disorders

We have previously reported that prenatal progestin exposure induces autism-like behavior in offspring through ERβ (estrogen receptor β) suppression in the brain, indicating that progestin may induce autism spectrum disorders (ASD). In this study, we aim to investigate whether prenatal progestin exposure is associated with ASD. A population-based case-control epidemiology study was conducted in Hainan province of China. The ASD children were first screened with the Autism Behavior Checklist (ABC) questionnaire, and then diagnosed by clinical professionals using the ASD diagnosis criteria found in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-V). Eventually, 235 cases were identified as ASD from 37863 children aged 0–6 years old, and 682 matched control subjects with typically developing children were selected for the analysis of potential impact factors on ASD prevalence using multivariate logistic regression. Our data show that the ASD prevalence rate in Hainan was 0.62% with a boy:girl ratio of 5.4:1. Interestingly, we found that the following factors were strongly associated with ASD prevalence: use of progestin to prevent threatened abortion, use of progestin contraceptives at the time of conception, and prenatal consumption of progestin-contaminated seafood during the first trimester of pregnancy. All the above factors were directly or indirectly involved with prenatal progestin exposure. Additionally, we conducted in vivo experiments in rats to further confirm our findings. Either endogenous (progesterone) or synthetic progestin (norethindrone)-treated seafood zebrafish were used to feed pregnant dams, and the subsequent offspring showed autism-like behavior, which further demonstrated that prenatal progestin exposure may induce ASD. We conclude that prenatal progestin exposure may be associated with ASD development

Robust estimation of bacterial cell count from optical density

Optical density (OD) is widely used to estimate the density of cells in liquid culture, but cannot be compared between instruments without a standardized calibration protocol and is challenging to relate to actual cell count. We address this with an interlaboratory study comparing three simple, low-cost, and highly accessible OD calibration protocols across 244 laboratories, applied to eight strains of constitutive GFP-expressing E. coli. Based on our results, we recommend calibrating OD to estimated cell count using serial dilution of silica microspheres, which produces highly precise calibration (95.5% of residuals <1.2-fold), is easily assessed for quality control, also assesses instrument effective linear range, and can be combined with fluorescence calibration to obtain units of Molecules of Equivalent Fluorescein (MEFL) per cell, allowing direct comparison and data fusion with flow cytometry measurements: in our study, fluorescence per cell measurements showed only a 1.07-fold mean difference between plate reader and flow cytometry data